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Specification 

1 . Title of the invention 

METHOD FOR RETRIEVING STORED MULTIMEDIA INFORMATION 

2 . Claim 

1. A method for retrieving a stored multimedia 
information, characterized by the fact that in a multimedia 
information storage and exchange system that stores message 
information consisting of coded voices and images received from 
optional video telephone terminals connected to a digital 
telephone network in mailboxes in a center and transmits a 
retrieved message information to the above-mentioned video 
telephone terminals when there is a retrieval request of the 
message information in the above-mentioned mailbox from optional 
video telephone terminals, an optional coded image information 
for retrieval is extracted from the message information stored 
in the above-mentioned mailbox; the coded image information for 
retrieval extracted is decoded; the image information for 
retrieval decoded is sub -sampled according to a prescribed 
reduction ratio to reduce the image; only a prescribed message 
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information portion of said decoded image information is 
temporarily stored; the image information temporarily stored is 
drawn out and synthesized with the decoded image information for 
retrieval of one frame; a synthesized image information is 
prepared by encoding the image information synthesized and 
stored; if there is a retrieval request from an optional video 
telephone terminal, the above-mentioned synthesized image 
information for retrieval is drawn out and directly transmitted 
to the above-mentioned video telephone terminal; if the above- 
mentioned synthesized image information for retrieval does not 
exist, the above-mentioned reduced decoded image information is 
drawn out, synthesized with one frame, encoded to generate a 
synthesized image information for retrieval at real time, and 
transmitted to the above-mentioned video telephone terminal; 
said video telephone terminal selects and retrieves an optional 
message from several messages of the above-mentioned synthesized 
image information for retrieval. 

3 . Detailed explanation of the invention 
(Industrial application field) 

The present invention pertains to a method for retrieving a 
stored multimedia information that automatically prepares an 
image for retrieval at real time in a storage and information 
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exchange system of a message information consisting of voices 
and image information sent via a digital video telephone 
network, sends it to a user, rapidly retrieves the stored /2 
information, and can, read out a desired voice and image 
information. 
(Prior art) 

Image information supply type services that store image 
information in an information processing center connected to a 
digital telephone network and have access to the image 
information from each terminal via a telephone network have been 
provided. 

For example, a communication system (visual message dial 
system) that directly inputs a real-time image information by a 
user using INS net 64 and can freely drawn it out is developed. 
In the visual message dial system, a center that inputs, stores, 
and outputs messages consisting of images and a terminal of a 
video telephone are connected by INS net 64 . In an image 
transmission and a voice transmission, 64 Kbps is respectively 
used, and for example, a message with a face confirmation, a 
presentation using a figure, etc., are possible. 

In such a service, in order to have access the image 
information stored in the information processing center from the 
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terminal and read out a necessary image information, it is 
essential to have a good retrieval function. 

On the other hand, as one of the representative retrieval 
functions being provided to terminal users, there is a direct 
retrieval function that displays a list of title images in a 
reduced image shape by representing the contents of each stored 
image information and directly selects a desired access 
information from them. This function is very effective as a 
retrieval method in the case where a specific classification 
arrangement for retrieval is not carried out for stored 
information groups or in the case where a specific 
classification arrangement is not carried out since it cannot 
predict which information is interested by each user. 

However, in the retrieval by a list display of the above- 
mentioned reduced title images, since the image information 
being actually stored are separate images fore retrieval, images 
for retrieval must be separately prepared by the information 
processing center in order to implement the direction retrieval 
function. 

Also, in a videotex service which is a representative image 
information supply type service, it is a specific image provider 
that provides stored image information, and a general terminal 
user mainly draws out the image information being provided and 
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views or utilizes it- Therefore, in such a service, it is 
necessary to register the image information itself being 
provided in the information processing center by preparing 
images for retrieval in accordance with the image information 
being provided by the center and to register and store the 
images for retrieval . 

(Problems to be solved by the invention) 

Recently, along with the supply of digital communication 
networks, as terminals that utilize high speed -and high band 
characteristics of the digital communication networks, the 
development of image terminals such as digital video telephone 
and multimedia information storage and exchange services for 
exchanging multimedia messages consisting of images and voices 
has been advanced. For example, in * Proposal of Real-Time Image 
and Voice Information Storage and Exchange Service Method" 
described in pp. 1-113 of * Information Processing Society, 41 st 
National Convention Lecture Thesis Collection," a multimedia 
information storage and exchange service is mentioned. 

In the multimedia information storage and exchange service, 
the storage of image and voice messages from a terminal to a 
center and the transmission of the stored image and voice 
messages from the center to the terminal are simultaneously 
carried out between the information processing center connected 
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to a digital communication network and the image terminal such 
as digital video telephone. For this reason, a stored 
information retrieval method in which a direct retrieval is 
effective is used similarly to the above-mentioned image 
information supply type service. However, the above-mentioned 
image information supply type service is a service that stores 
image information in the center by an image provider, has access 
to the center by a general terminal, and retrieves and reads out 
a desired image information, and the center has a time margin 
until the provided images are registered after receiving them. 
On the contrary, the multimedia information storage and exchange 
service is a service that stores and exchanges the transmission 
and reception of image and voice messages being carried out 
between each image terminal, and the exchange has no time margin 
until the image information are stored and transmitted after 
receiving them. 

As a result, in the multimedia information storage and 
exchange service, since the image and voice messages being sent 
at each optional time from the image terminal such as digital /3 
video telephone are stored, there is a no margin for preparing 
images for retrieval in advance and storing them .in the 
information processing center, unlike the image information 
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supply type service. Therefore, the former method cannot be 
applied to the latter method. 

Also, the image information from the digital video 
telephone are usually compressed and encoded, transmitted onto a 
line, and stored in an encoded state in the center. Therefore, 
in case the above-mentioned direct retrieval is implemented in 
the center, it is necessary to implement other image processing, 
not to mention decoding and encoding of images, so that the 
response time for transmitting the image information for 
retrieval to the terminal is lengthened, thereby increasing the 
center load. 

Furthermore, a method that transmits several image 
information to video telephone terminals in advance and displays 
on a multi-screen at the terminal is also considered, however in 
this case, a video signal decoder with a high compression rate 
is required, so that the function increase at the terminal is in 
demand . 

The purpose of the present invention is to solve these 
conventional problems and to provide a method for retrieving a 
stored multimedia information that can prepare images for 
retrieval at real time from image message information being sent 
from an image terminal such as digital video telephone and does 
not require a function increase at the terminal, since a direct 
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retrieval function is realized by a multimedia information 
storage and exchange service . 
(Means to solve the problems) 

In order to achieve the above-mentioned purpose, the method 
for retrieving a stored multimedia information is characterized 
by the fact that an optional coded image information for 
retrieval is extracted from the message information stored in 
the mailbox; the coded image information for retrieval extracted 
is decoded; the image information for retrieval decoded is sub- 
sampled according to a prescribed reduction ratio to reduce the 
image; only a prescribed message information portion of said 
decoded image information is temporarily stored; the image 
information temporarily stored is drawn out and synthesized with 
the decoded image information for retrieval of one frame; a 
synthesized image information is prepared by encoding the image 
information synthesized and stored; if there is a retrieval 
request from an optional video telephone terminal, the 
synthesized image information for retrieval is drawn out and 
directly transmitted to the video telephone terminal; if the 
synthesized image information for retrieval does not exist, the 
reduced decoded image information is drawn out, synthesized with 
one frame, encoded to generate a synthesized image information 
for retrieval at real time, and transmitted to the video 
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telephone terminal; said video telephone terminal selects and 
retrieves an optional message from several messages of the 
synthesized image information for retrieval. 
(Operation) 

In the present invention, using the image information among 
the message information (coded voices and image information) 
being sent to the center from the information provider (digital 
video telephone) , images for retrieval at a time of the 
retrieval of these message information are prepared. In the 
conventional method, each time information is provided, images 
for retrieval have been manually renewed. On the contrary, in 
the present invention, when information are provided, they are 
automatically renewed at real time, and images for retrieval are 
prepared. 

In other words, in the center in the present invention, 
when the message storage from the terminal is completed, an 
appropriate partial image information of the messages is 
extracted, and the image is decoded by the decoder and sub- 
sampled at the compression ratio corresponding to the number of 
screen split of the image information for retrieval. The number 
of picture element is reduced, and the images are stored in a 
temporary storage for the image information for retrieval. At 
the same time, the reduced images for retrieval generated by the 
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processing, the reduced images for retrieval already stored in 
the temporary storage, and if necessary, the reduced images of 
dummy images are synthesized in a two-dimensional planar shape, 
so that an image information for retrieval of a multi- screen is 
generated, encoded by the image encoder, and stored in the 
storage device for the image information for retrieval- In this 
case, a terminal user may not be aware of the processing during 
the storage for postprocessing of the message storage, and since 
the processing for generating the images for retrieval of the 
latter-half multi -screen can be implemented in an empty time of 
the center processing, the operation in accordance with the load 
state at the center is possible. 

On the other hand, when a retrieval is requested from the 
terminal, the center transmits an image - information for 
retrieval already generated, implements an image information /4 
generation processing for retrieval at real time when there is a 
non-generated image information for retrieval, stores it, and 
transmits it to the terminal. In almost all the cases, since 
the coded image information for retrieval is only transmitted, a 
terminal user can immediately view the retrieving screen during 
the retrieval. In the worst case, though the time for 
generating one image screen for retrieval is required, since 
only the processing time for the synthesis of the multi-screen 
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and the encoding is required, a high-speed processing is 
possible. 

(Application example) 

Next, an application example of the present invention is 
explained in detail by the figures. 

Figure 1 is an entire constitutional diagram showing a 
stored multimedia information retrieval system which is an 
application example of the present invention. 

In Figure 1, 11-1L are optional video telephone terminals 
being connected to a digital telephone network, 21-2L are 
digital subscriber lines, 5 is a digital telephone network, 10 
is a digital subscriber line, 20 is a center, and 21 is an audio 
image information storage part. 

In the center 20, in order to generate an image for 
retrieval at real time from the image message information being 
sent from the image terminals 11 -1L such as digital video 
telephone, a specific one screen of the image message is 
automatically extracted as a title image. In order to generate 
the image for retrieval from the automatically extracted title 
image, a realizing method at the center 2 0 is used in terms of 
versatility, etc. In other words, the corresponding image 
encoder and decoder built in a minimum digital video telephone 
are mounted at the center 20, and an image decoding and decoding 
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processing with a large load is distributed, and other 
processing are applied using a processing empty time of the 
center, so that a high-speed responsivity to the terminals 11-1L 
and an efficient processing at the center 2 0 are implemented- 

Also, all the video telephone terminals have the same 
constitution and are provided with voice and image transmission 
and reception functions, the same screen resolution, and dial 
information transmission and reception functions- The center 20 
is provided with the same voice and image encoding and decoding 
functions and dial information transmission and reception 
functions as those of the video telephone terminals 11- 1L, and 
several mail boxes are opened in the internal audio image 
information storage part 21. 

Figure 2 is a constitutional diagram showing the center in 
Figure 1 • 

In Figure 2, 3 0 is a line control part provided with a 
terminal function of a digital telephone network, 31 is an audio 
image information storage part for storing coded voice and image 
message information being sent from the video telephone 
terminals 11-1L, 32 is an image information extraction part that 
extracts an optional partial image information from the message 
information stored in the audio image information storage part 
31, 33 is a decoding part for decoding the extracted image 
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information, 34 is an image information sampling part for 
sampling a decoded image information, 35 is a sampled image 
storage part for storing n-1 screens of the sampled image 
information when an image for retrieval is a n split screen, 36 
is an image synthesis part for synthesizing n screens of the 
sample screens, 37 is an encoding part for encoding the image 
synthesis information, 3 8 is a retrieved image storage part for 
storing the synthesized encoded image information for retrieval, 
and 3 9 is a retrieval control part for controlling the audio 
image information storage part 31, image information extraction 
part 32, sampled image storage part 35, image synthesis part 36, 
and retrieved image storage part 38 during processing from each 
video telephone terminals 11 -1L. 

The operations of Figures 1 and 2 are mentioned. 

An optional mailbox i is designated from an optional video 
telephone terminal 11, and a communication information 50 is 
connected via the digital telephone network 5 and transmitted to 
the line control part 3 0 of the center 20. 

In the center 20, the communication information 50 meaning 
the designation of the mailbox i received from the video 
telephone terminal 11 is separated into a voice and image 
message information 51 and a key input information 52 in the 
line control part 30, and the former is transferred to the audio 
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image information storage part 31 and stored. The latter 
decides the end of the voice and image message information 
destined to the mailbox i in the retrieval control part 39. 
When it is decided that the voice and image message destined to 
the mailbox i is finished, the retrieval control part 39 inputs 
an optional partial image information extraction instruction 70 
from the stored message information in the mailbox i into the 
image information extraction part 32 . The image information 
extraction part 32 extracts an optional image from the voice and 
image information storage part 31 and inputs an extracted /5 
image information 53 into the decoding part 33 . The decoding 
part 33 decodes the extracted image information input from the 
image information extraction part 32 and inputs a decoded image 
information 54 into the image information sampling part 34. The 
image information sampling part 34 samples the decoded image 
information 54 in accordance with the size of a reduced screen 
being displayed at the terminal during the retrieval, reduces 
the amount of information, and stores a sampled image 
information 55 converted into a reduced image. In this case, 
the information are stored up to the amount of information of n- 
1 for the number n of reduced images being displayed on one 
screen, and when a new sampled image is stored or if the amount 
of information of one screen is attained, the sampled image 
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information 55 is input into the image synthesis part 36. The 
image synthesis part 36 synthesizes the input sampled image 
information 55 on one screen and inputs the image synthesis 
information 56 into the encoding part 37. The encoding part 37 
compresses and encodes the image synthesis information 56 and 
inputs the image information 57 for retrieval into the retrieved 
mage storage part 38. The retrieved information storage part 38 
stores the compressed encoded image information 57 for 
retrieval . 

Next, a method that retrieves a message information in the 
mailbox i and reads it out is mentioned. 

An optional video telephone terminal 13 is connected to the 
center 20, and the communication information 50 of the video 
telephone terminal 13 is transferred as the key information 52 
via the line control part 30 of the center 2 0 from the line 
control part 3 0 to the retrieval control part 39. For a 
retrieval request instructed from a terminal user by the key 
information 52, the retrieval control part 39 reads out the 
retrieved image storage part 38 as to whether or not the image 
information 57 for retrieval for the corresponding mailbox is 
stored and transfers the retrieved image information 57 to the 
line control part 30. If it is decided that the corresponding 
information does not exist, the sampled image information 55 of 
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the sampled image storage part 35 is further retrieved, and a 
retrieval result information 72 is decided. If it is decided 
that the corresponding information exists in the sampled image 
storage part 35, an image synthesis instruction 73 of the 
sampled image information 55 of the sampled image storage part 
35 is sent to the image synthesis part 36- The image synthesis 
part 3 6 synthesizes the sampled image information to one image, 
encodes it by the encoding part 37, and transfers the image 
information 57 for retrieval to the retrieval control part 39. 
After receiving the transfer, the retrieval control part 39 
transfers the image information 57 for retrieval to the line 
control part 30. The line control part 30 transmits the image 
information 57 for retrieval to the video telephone terminal 13 
and waits for the next communication information 5 0 from the 
video telephone terminal 13. If the next communication 
information 50 is input into the retrieval control part 3 9 via 
the line control part 30, when it is a retrieval request, the 
same processing as the above-mentioned information retrieval 
request is carried out. Also, for an information readout 
request, the retrieval control part 3 9 sends a readout 
instruction 74 of the voice and image message information 51 
stored in the voice image information storage part 31. The 
voice and image information storage part 31 transfers the voice 
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and image message information 51 to the line control part 30. 
The line control part 30 outputs a message to a display and a 
telephone set of the video telephone terminal 13 by transmitting 
the transferred voice and image message information 51 to the 
video telephone terminal 13 . 

After receiving the image information for retrieval, the 
video telephone terminal 13 selects an optional message from the 
image information of several messages once displayed in one 
screen and can retrieve a desired message. 

First, if an optional mailbox is designated from the video 
television terminal and a voice and image information is sent 
(step 101) , the end of the information is decided (step 102) , 
and the extraction of an optional partial image information from 
the stored message information is instructed (step 103) . Thus, 
the image information extraction part 32 extracts an image as a 
retrieval image (step 104) , decodes it (step 105) , samples it in 
accordance with the size of a reduced screen for displaying it 
at the terminal (step 106), and stores it. Next, the input 
sampled mage information is synthesized with a screen (step 107) 
and further compressed and encoded, so that an image information 
for retrieval is generated (steps 108 and 109) . 

If a retrieval request from the video telephone terminal 
exists (step 201) , the image information for retrieval is 
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retrieved (step 2 02) , and whether or not the information exists 
is decided (step 203) . If the image information exists, it is 
read out and transmitted to the video telephone terminal from /6 
which the request is made (step 2 04) . Also, there is no image 
information for retrieval, an image information for retrieval is 
generated at real time from the sampled image storage part 35. 
For this purpose, the sampled image information is retrieved 
(step 2 05) , and whether or not the information exists is decided 
(step 2 06) . If the information exists, an image synthesis 
instruction is output (step 207), and the sampled image 
information is synthesizes with a screen (step 208) . Then, it 
is encoded (step 209) . The image information for retrieval is 
transmitted to the terminal from which the request is made (step 
210) . Then, the next reception from the terminal is waited 
(step 211) . If there is a reception, the flow returns to step 
101 or 201. 

Thus, in this application example, even in a multimedia 
information storage and exchange service, when there is a 
retrieval request from the terminal, the image information for 
retrieval already generated is transmitted to the center, and if 
the image information is not generated yet, an image information 
is generated at real time and stored, so that a terminal user 
can immediately view a retrieving screen during the retrieval. 
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(Effects of the invention) 

As explained above, according to the present invention, an 
image information for retrieval is sent to a video telephone 
terminal, and a desired readout information from a multi- screen 
is selected by the video telephone terminal, so that the 
information stored in the center can be read out at random, 
thereby being able to shorten the retrieval time- Furthermore, 
since the message information is displayed on the multi-screen 
of the video telephone terminals, the contents of partial of 
various message information can also be detected. 

4 . Brief description of the figures 

Figure 1 is an entire constitutional diagram showing a 
stored multimedia information retrieval system which is an 
application example of the present invention. Figure 2 is a 
constitutional diagram showing a center in Figure 1. Figure 3 
is an operational flow chart showing a retrieved image 
generation and a retrieval readout processing in the center. 
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Video telephone terminal 
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Digital telephone network 
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Figure 2 : 

Line control part 

Audio image information storage part 
Image information extraction part 
Decoding part 

Image information sampling part 
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Figure 3 : 
A, Start 
B* End 

101 Reception of an audio image message designated by a 
mailbox? 

102 Decision of message end? 

103 Extraction instruction of partial image information 

104 Extraction of image 

105 Decoding 

106 Sampling at a reduced screen size 

107 Synthesis of sampled image 
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108 Compressing and encoding 

109 Storage of retrieved image 

201 Existence of a retrieval request from a terminal 

2 02 Retrieval of retrieved image storage part 

203 Existence of a retrieved image? 

204 Transmission to terminal 

205 Sampled image retrieval 

2 06 Existence of a sampled image? 

2 07 Image synthesis instruction 

208 Synthesis 

2 09 Encoding 

210 Transfer of an image for retrieval 

211 Wait for the next reception 
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